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SUMMARY 


Problem 

The  purpose  of  this  study  was  to  take  an  in-depth  look  at  the  Air  Force  field  evaluation  process  and 
to  develop  a methodology  for  improved  data  analysis  and  reporting.  It  was  felt  that  the  present  analysis 
system  could  be  expanded  to  provide  a much  more  meaningful  report  for  course  personnel. 

(Approach 

Field  evaluation  data  for  two  resident  training  courses,  Inventory  Management  (IM)  (3ABR64530) 
and  Materiel  Facilities  (3ABR64730)  was  analyzed  into  a number  of  cohesive  task  clusters  using  factor 
analysis.  These  clusters  were  then  related  to  graduates’  duty  type,  work  environment  and  type  of  activity. 

Results 

Each  of  the  task  clusters  was  analyzed  in  terms  of  the  average  specialty  training  standard  (STS) 
requirement  and  supervisor  rating,  as  well  as  average  percent  performed  (average  performance  rate  for  tasks 
in  a cluster)  and  percent  below  standard  (as  defined  in  the  STS).  Training  adequacy  was  evaluated  by 
utilizing  a training  index  which  was  the  difference  between  average  STS  and  average  supervisor  rating  of  a 
task  cluster.  A number  of  IM  task  clusters  were  found  to  be  undertrained,  while  a number  of  clusters  in 
each  specialty  were  found  to  be  overtrained. 

Conclusions 

Suggestions  for  improving  the  field  evaluation  system  were  (a)  increased  flexibility  in  data  analysis 
capability,  (b)  feedback  to  the  courses  should  be  specific  and  meaningful,  and  (c)  emphasis  should  be 
placed  on  task  cluster  reporting,  as  well  as  individual  task  reporting. 
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APPRAISAL  OP  AIR  FORCE  TRAINING  COURSE 
FIELD  EVALUATION  SYSTEM 

L INTRODUCTION 


Technical  training  is  primarily  vocational  education.  It  includes  virtually  all  Air  Force  training  other 
than  basic  military  training,  aircrew  training,  and  professional  education.  Most  technical  training  is  achieved 
through  organized  on-the-job-training  (OJT)  programs  or  in  resident  courses  conducted  at  the  five  technical 
training  centers  of  the  Air  Training  Command.  In  order  to  scope  the  technical  training  enterprise,  the  Air 
Training  Command  alone  conducts  approximately  1,000  different  resident  courses  and  graduates  about 
180,000  students  annually.  These  courses  run  the  gamut  from  relatively  mundane  service  functions  (e.g., 
protective  coatings  specialist)  to  exotic  high  technology  (e.g.,  computer  systems  analyst).  An  important 
step  in  die  training  process  is  to  evaluate  the  extent  to  which  resident  training  is  meeting  the  needs  of  the 
user;  Air  Fora1  field  evaluation  was  designed  to  accomplish  this  goal. 


Present  Field  Evaluation  System 

The  present  field  evaluation  system  is  an  integral  part  of  the  instructional  systems  development  (ISD) 
model  employed  by  the  Air  Force.  Field  evaluation,  along  with  internal  evaluation,  constitutes  Step  5 of 
the  ISD  model  (see  Figure  1).  Field  evaluation  is  designed  to  ensure  that  job  performance  requirements  are 
met  by  graduates  and  to  determine  supervisors’  and  graduates'  opinions  of  the  training  program. 

In  the  present  system,  a course  is  evaluated  based  on  questionnaires  and  field  visits.  Two  types  of 
questionnaires  are  generally  used.  One  is  for  the  graduate  supervisor;  the  other  is  tor  the  course  graduate  In 
most  cases,  the  questionnaire  is  designed  for  completion  by  the  graduate  immediate  supervisor. 

Field  visits  are  performed  by  personnel  from  the  Training  Evaluation  Division.  They  at  temp;  to  visit  a 
representative  sampling  of  graduates  3 to  6 months  after  the  graduates'  assignment  to  the  job.  Data  is 
gathered  through  observation  and  interviews  with  the  graduate  and  Itis  supervisor. 


In  the  present  field  evaluation  system,  the  only  data  analysis  available  is  in  ATC  Manual  52-334.  The 
steps  from  designing  the  questionnaire  to  getting  the  data  into  the  computer  to  interpreting  the  results  are 
covered  here.  A statistical  data  analysis  package  providing  analysis  flexibility  is  not  provided  for  use  by 
course  evaluators.  The  associated  computer  program  has  minimal  analysis  power  providing  only  frequencies 
and  percentages  by  questionnaire  item.  The  existing  system  merely  provided  a frequency  count  of  responses 
to  questionnaire  items.  The  most  thoroughly  analyzed  items  arc  the  task  proficiency  ratings  for  each 
graduate.  This  analysis  consists  of  computing  by  task  the  percentage  of  latings  which  meet  or  exceed  the 
STS  requirement.  If  this  percentage  does  not  meet  the  established  criterion  of  80%,  the  task  was  considered 
to  have  not  been  sufficiently  trained  in  the  course. 

Goal  of  Effective  Field  Evaluation 

Effective  field  evaluation  of  graduates  can  be  thought  of  as  the  quality  control  step  in  the  ISD 
process.  The  field  evaluation  of  graduates  should  determine  how  well  the  course  prepared  students  to 
satisfy  job  performance  requirements  by  identifying  strengths  and  weaknesses  in  the  student's  job 
performance.  In  addition,  it  should  indicate  how  the  course  might  be  improved  in  order  to  correct 
deficiencies  in  job  performance.  In  essence,  field  evaluation  data  becomes  a significant  source  of  feedback 
to  ISD  personnel.  Analysis  of  the  field  evaluation  data  should  identify  the  strengths  and  weaknesses  in  a 
course.  Furthermore,  the  field  evaluation  report  should  reflect  these  strengths  and  weaknesses,  and  provide 
input  to  the  solution  of  problems. 

Another  aspect  of  effective  field  performance  which  is  rarely  considered  is  the  graduate's  ability  to 
upgrade.  If  technical  training  does  not  provide  well  trained  graduates  who  are  able  to  effectively  upgrade, 
the  system  is  not  fulfilling  its  training  obligation.  Data  on,  time  to  upgrade,  career  development  course 
(CDC),  and  specialty  knowledge  test  (SKT)  scores  should  constitute  a routine  part  of  a field  evaluation. 
These  data  should  be  continually  tracked  in  order  to  monitor  changes  which  occur  in  the  field. 
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Figure  1.  Major  activities  and  decisions  of  the  evaluation  process. 
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Limitations  of  Field  Evaluation 

There  is  an  inevitable  compromise  required  when  attempting  to  satisfy  the  goals  of  effective  field 
evaluation.  This  compromise  is  between  the  validity  of  the  evaluation  system,  in  terms  of  how  effectively 
the  results  of  an  evaluation  bear  on  training  decisions,  and  the  cost  of  such  an  evaluation.  Perhaps  an  ideal 
evaluation  mechanism  would  involve  “capturing”  the  graduate  immediately  after  arrival  at  his  duty 
assignment  and  administering  an  objectively  scored  job  sample  test.  The  goal  of  such  a test  would  be  to 
accurately  assess  the  state  of  graduate  proficiency  in  light  of  actual  STS  requirements.  Unfortunately,  such 
an  approach  would  be  extremely  costly. 

Because  of  the  prohibitive  cost  of  large-scale  job  sample  testing,  the  Air  Force  uses  a questionnaire 
based  supervisor/graduate  rating  system  for  field  evaluation.  Supervisors  are  asked  to  rate  the  graduate’s 
proficiency  on  a series  of  tasks  taken  from  the  STS.  However,  the  potential  limitations  associated  with  any 
rating  procedure  are  well  known.  For  example,  one  must  make  assumptions  concerning  rater  frame  of 
reference  across  ratings,  rater  reliability,  as  well  as  a variety  of  possible  bias  effects.  These  problems  are 
further  confounded  by  the  contaminating  influence  of  OJT.  That  is,  OJT  as  well  as  resident  training 
contributes  to  the  proficiency  of  the  graduate  at  the  time  of  rating  by  the  supervisor.  Other  factors,  such  as 
motivation,  also  affect  the  proficiency  ratings  a graduate  receives. 

Until  alternate  procedures  are  devised,  we  should  seek  new  methods  and  procedures  to  make  the 
existing  evaluation  system  as  meaningful  as  possible.  The  data  to  be  presented  below  indicate  a striking 
degree  of  consistency  in  supervisor  ratings  which  suggests  that  most  supervisors  have  been  attempting  to 
make  accurate  and  conscientious  judgments  regarding  graduate  proficiency.  This  consistency  of  ratings 
contradicts  the  frequent  allegations  concerning  the  lack  of  reliability  of  field  followup  data.  Furthermore, 
this  consistency  indicates  that  we  need  to  achieve  more  comprehensive  reporting  and  integration  of  field 
data  rather  than  a complete  revamping  of  the  system. 

Training  Evaluation  in  Multiple  Job  Specialties 

Training  Evaluation  is  also  complicated  by  the  fact  that  Air  Force  specialties  are  not  equally 
homogeneous.  That  is,  in  some  fields  it  may  be  fair  to  assume  that  a large  number  of  tasks  will  be 
performed  by  all  graduates.  In  other  fields  where  graduates  may  be  assigned  to  different  duty  sections  and 
specialize  in  different  activities,  this  assumption  may  not  be  valid.  Graduate  evaluation  surveys  must  take 
degree  of  homogeneity  within  specialties  into  account.  It  is  difficult  to  draw  conclusions  from  the  finding 
that  a low  percentage  of  graduates  perform  a given  task  without  knowing  whether  this  task  is  infrequently 
performed  by  all  graduates  in  a field,  or  whether  it  is  frequently  performed  by  a subset  of  graduates  and 
never  performed  by  the  rest.  In  a field  which  is  relatively  homogeneous,  a resident  course  may  train  on  only 
those  tasks  which  are  performed  by  high  percentage  of  graduates.  In  a more  differentiated  field,  the 
question  of  whether  training  should  be  provided  on  a task  cannot  be  resolved  by  reference  to  a single 
percentage. 

Thus,  it  is  necessary  for  graduate  evaluations  to  take  degree  of  homogeneity  into  account.  To  do  this, 
it  is  best  if  Occupational  Survey  Reports  (OSR)  are  used  to  help  design  evaluation  questionnaires  by 
showing  what  task  clusters  are  to  be  anticipated.  By  drawing  on  the  OSR,  the  evaluator  can  make  sure  that 
the  sample  of  tasks  on  the  evaluation  questionnaire  adequately  represents  all  major  jobs  performed  by 
graduates  in  the  field. 

Graduate  evaluation  surveys  and  occupational  surveys  should  complement  one  another.  Since 
occupational  surveys  are  performed  but  once  every  several  years,  they  may  become  out-of-date.  Graduate 
evaluations  offer  the  opportunity  to  reassess  the  findings  of  an  occupational  survey  with  respect  to  the 
more  limited  population  of  technical  training  graduates.  Congruency  between  findings  from  the  two  types 
of  surveys  also  strengthens  the  evaluation  survey,  which  does  not  have  the  advantage  of  the  comprehensive 
task  listings,  extensive  sampling,  and  controlled  administration  that  characterize  the  occupational  survey. 

The  inventory  management  (IM^  and  materiel  facilities  (ME)  specialties  are  examples  of 
heterogeneous  career  ladders.  The  OSR1  on  5,544  airmen  in  these  two  fields  produced  the  following 
conclusions: 


'Occupational  Survey  Report.  Occupational  Survey  Branch,  37000  OCM  SO.  Lackland  Al'B.TX:  I October  1972. 
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1 . The  1M  and  Ml;  career  ladders  at  that  time  were  made  up  of  2d  clusters  of  tasks. 

2.  While  the  career  ladders  were  thus  extremely  heterogeneous,  the  clusters  were  internally 
homogeneous. 


3.  Transfer  of  airmen  between  jobs  within  career  ladders  was  rare. 

4.  The  probability  that  an  airman  in  his  first  job  assignment  would  perform  any  given  task  was  low. 

Such  findings  pose  serious  issues  for  training  and  personnel  policy.  From  the  point  of  view  of  tire 
training  organization,  it  is  not  cost  effective  to  provide  resident  training  on  tasks  which  most  graduates  will 
not  perform.  Yet,  to  do  otherwise,  may  have  serious  consequences  for  personnel  assignments,  utilization, 
and  upgrading.  Upgrading  requirements  as  previously  mentioned  are  an  important  factor  which  resident 
training  cannot  ignore.  The  test  requirements  established  by  the  Weighted  Airman  Promotion  System 
(WAPS)  become  an  additional  criterion  of  field  performance  that  must  be  considered.  If  a WAPS  test  covers 
a variety  of  jobs,  most  of  which  the  individual  airman  will  not  have  the  opportunity  to  perform,  then  an 
efficient  training  strategy  must  be  devised  for  airmen  to  acquire  knowledge  about  jobs  they  do  not  perform. 
To  see  whether  graduates  are  in  fact  able  to  meet  upgrading  requirements,  data  on  OJT  and  SKT 
performance  should  constitute  a routine  part  of  field  evaluation. 

In  dealing  with  a multiple  job  field,  four  training  strategies  might  be  considered.  It  should  be  noted 
that,  to  make  an  intelligent  choice  among  these  strategies,  the  consequences,  not  just  for  training  but  for 
the  total  personnel  system,  must  be  taken  into  account. 

Resident  Training  for  a Variety  of  Jobs.  This  strategy  would  attempt  to  train  airmen  for  the  various 
jobs  making  up  a specialty,  even  though  the  probability  that  a graduate  will  have  to  perform  any  given  job 
is  small.  From  the  point  of  view  of  personnel  policy,  this  strategy  has  a number  of  advantages.  By  creating  a 
large  pool  of  airmen  with  diverse  competencies,  it  facilitates  assignment  and  utilization.  This  strategy  also 
gives  first-term  airmen  a broader  view  of  an  Air  Force  occupation.  This  broader  perspective  may  motivate 
airmen  and  help  them  perform  more  competently,  although  this  contention  has  not  been  proved.  This 
strategy  may  also  give  airmen  a head  start  in  preparing  for  WAPS  tests.  These  advantages  are 
counterbalanced  by  the  high  training  costs  incurred  when  airmen  are  trained  on  jobs  they  will  not  perform. 

OJT  (Directed  Duty  Assignments).  An  alternative  is  to  forego  resident  training  and  depend  solely 
upon  OJT.  This  policy  assumes  that  the  skills  involved  can  be  learned  without  too  much  difficulty  or  cost 
in  the  field.  The  argument  for  OJT  is  also  strengthened  if  one  assumes  that  knowledge  about  how  other  jobs 
are  performed  contributes  little  to  proficiency  in  an  airman’s  first  job.  However,  a broader  perspective 
might  be  incorporated  into  this  strategy  by  requiring  systematic  rotation  of  first-term  airmen  among  a 
variety  of  related  jobs. 

Resident  or  Field  Training  on  Prototypical  Tasks.  A compromise  between  comprehensive  resident 
tr  lining  and  complete  reliance  on  OJT  would  be  to  select  prototypical  tasks,  which  would  facilitate  the 
learning  of  other  skills  not  formally  taught.  Some  tasks  might  be  taught  because  they  are  common  to  a 
number  of  jobs  within  the  specialty.  Others  might  be  performed  within  only  one  job,  but  nonetheless 
provide  experience  and  knowledge  wliich  would  accelerate  training  on  other  tasks  for  which  no  training  is 
provided.  Fxamples  from  the  1M  specialty  might  include  the  operation  of  remote  terminals  and  searches  of 
various  reference  publications. 

Resident  Training  for  a Single  Job.  Front  the  training  point  of  view  it  would  be  most  effective  to  train 
for  a single  job  in  the  field.  However,  this  strategy  requires  changes  in  personnel  policy  which  would  insure 
that  an  airman  would  be  assigned  to  the  jobs  for  which  he  had  been  trained  and  that  he  could  be  upgraded 
without  knowledge  of  other  jobs. 


II.  RATIONAL!-’  AND  MKTIIOD 

: 

Data  Base  and  Instruments 

Field  evaluation  data  from  two  course  surveys,  originally  carried  out  by  the  Lowry  Training 
Evaluation  Division,  were  reanalyzed  for  this  study: 

I.  IM  course,  3ABR64530.  Information  concerning  students  who  graduated  from  1 August  1975 
through  17  September  1975  was  obtained  from  questionnaires  mailed  from  14  November  1975  through  31 
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December  1975.  Forty-six  job  tusks  were  selected  from  Specialty  Training  Standard  645X0,  dated  4 June 
1974.  A total  of  163  usable  surveys  were  returned  out  of  the  208  that  were  mailed. 


2.  MF  course,  3ABR64730.  This  survey  was  completed  during  November  and  December  of  1975. 
Thirty-two  job  tasks  were  selected  from  Specialty  Training  Standard  647X0,  dated  30  May  1974. 
Sixty-nine  usable  surveys  were  returned  out  of  103  mailed. 

Of  interest  on  each  survey  were  four  types  of  variables:  (a)  the  graduate's  duty  assignment  within  the 
Chief  of  Supply  organization,  (b)  the  graduate's  type  of  work  environment,  (c)  a classification  of  the  type 
of  activities  which  the  graduate  engaged  in  during  his  current  assignment,  and  (d)  ratings  of  job  tasks 
selected  from  the  STS.  These  variables  are  listed  in  Appendices  A and  B.  These  job  task  ratings  consisted  of 
the  supervisor’s  appraisal  of  the  student’s  performance  on  a scale  from  0 to  5.  A rating  of  0 corresponded  to 
“incapable  of  performing,”  1 to  “extremely  limited,”  2 to  "partially  proficient,"  3 to  “competent,”  4 to 
“highly  proficient,”  and  5 to  “not  performed.” 

Defining  Task  Clusters 

In  order  to  relate  graduate  performance  to  the  various  jobs  in  the  MF  and  IM  fields,  the  task  ratings 
were  used  to  derive  a group  of  task  dimensions  or  clusters  for  each  specialty  and  then  these  clusters  were 
characterized  in  terms  of  type  of  duty,  activity,  and  work  variables.  These  latter  variables  served  to  define 
jobs  (i.e..  clusters  of  tasks)  or  parts  of  jobs  which  the  sample  of  graduates  found  themselves  performing 
after  graduation.  It  was  the  intention,  then,  to  evaluate  training  at  the  general  task  cluster  level  rather  than 
the  specific  task  level.  It  was  hypothesized  that  this  approach  would  lend  considerable  clarity  to  field 
evaluation  reporting,  and  would  have  many  implications  for  training  modification. 

As  mentioned  earlier,  each  graduate  was  rated  as  “not  performing”  (5)  or  on  a continuum  of  quality 
of  performance  (0  4)  for  those  tasks  listed  on  the  field  questionnaire.  Initially,  this  data  was  regarded  as  a 
binary  classification  of  tasks  into  categories  of  “performed”  or  “not  performed”  for  each  graduate.  Thus, 
each  task  was  scored  as  a 1 if  it  was  given  a rating  of  0 4 and  scored  it  0 if  the  rating  was  5. 

Included  in  this  first  analysis  was  a set  of  new  variables  corresponding  to  type  of  duty  (listed  in 
Appendix  A).  For  each  specialty  (i.e.,  IM  or  MF)  there  was  list  of  duty  areas  in  which  the  graduate 
performed.  The  supervisor  was  asked  to  check  the  duty  area  in  which  the  graduate  worked.  Duty  variables 
were  then  created  by  scoring  1 if  the  graduate  was  checked  as  being  in  this  duty  area,  and  0 otherwise. 

These  duty  variables  were  combined  with  the  binary  task  variables,  intercorrelated  and  factor 
analyzed.  Factor  analysis  is  a procedure  whereby  clusters  of  correlated  variables  are  identified  which  define 
a more  global  variable  that  we  shall  call  a task  cluster.  The  variables  defining  each  task  cluster  can  be 
deduced  from  a factor  loading  matrix.  All  variables  having  a high  factor  loading  on  a particular  factor  are 
combined  conceptually  into  a task  cluster. 

At  this  point,  factor  scores  for  each  graduate  on  each  of  the  factors  are  computed.  Factor  scores  can 
be  regarded  as  similar  to  scores  on  any  test.  A high  factor  score  says  that  a graduate  scored  high  on  the 
cluster  of  tasks  defining  the  factor.  In  this  case,  a high  score  indicates  that  those  tasks  the  graduate  was 
rated  as  performing  were  very  much  like  those  tasks  defining  the  factor.  Thus,  we  used  the  factor  scores  to 
classify  graduates  as  either  belonging  to  or  not  belonging  to  a particular  task  cluster.  Some  of  the  graduates 
did  not  score  high  on  any  of  the  task  clusters.  This  indicated  either  that  the  questionnaire  did  not  include 
enough  of  the  tasks  that  certain  graduates  were  expected  to  perform  on  the  job  or  that  some  graduates 
performed  unique  configurations  of  the  tasks  on  the  questionnaire. 

Relating  Task  Clusters  to  Other  Variables.  Once  this  initial  analysis  was  performed,  we  could  ask  how 
other  classifying  variables  related  to  the  identified  task  clusters.  The  classifying  variables  we  were  interested 
in  were:  (a)  type  of  work  environment,  and  (b)  general  types  of  work  activities  performed  by  graduates 
(Appendix  A).  This  produced  an  additional  set  of  32  binary  variables  indicating  “performed”  or  “not 
performed.”  Each  of  these  variables  was  related  to  an  original  task  cluster  in  the  sense  of  how  much  each 
variable  correlated  with  each  task  cluster.  This  proved  to  be  another  aid  in  defining  the  nature  of  the 
particular  job. 
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III.  KKSULTS 


Mean  Level  of  Performance  by  Task  and  Duty 

As  a first  step  in  the  analysis,  means  were  computed  for  all  the  binary  variables  to  be  factor  analyzed. 
These  means  are  listed  in  Appendices  C and  I),  under  the  factors  to  which  the  variables  were  ultimately 
assigned.  Each  mean  reflects  the  proportion  of  graduates  in  the  sample  who  performed  the  task  in  question. 
For  example.  Task  1 (T 1 ) has  a mean  of  .17  in  the  1M  sample; in  other  words,  17%  of  the  graduates  in  the 
sample  performed  this  task. 

Table  1 summarizes  these  computations  by  showing  the  number  of  tasks  at  different  rates  of 
performance.  It  shows  that  17  of  the  46  tasks  on  the  IM  questionnaire  were  performed  by  less  than  10%,  of 
the  graduates.  In  general,  the  incidence  of  task  performance  was  low,  particularly  in  IM  where  only  three 
tasks  were  performed  by  more  than  40%  of  the  graduates.  These  results  were  consistent  with  the  results  of 
the  occupational  survey  of  1972.  In  that  survey,  the  task  most  frequently  performed  by  airmen  in  their  first 
assignment  was  to  operate  remote  keyboard  printers.  This  task  was  performed  by  32%  of  such  airmen.  The 
same  task  was  tied  for  the  most  frequently  performed  task  in  the  graduate  evaluation  sample,  with  49%  of 
the  graduates  performing  the  task.  These  data  indicated  low  utilization  of  training,  at  least  as  far  as  direct 
task  performance  was  concerned. 


Table  1.  Distribution  of  Tasks  by 
Percentage  of  Graduates  Performing 


Number  of  Tasks 

% Performing  Task 

Inventory 

Management 

Materiel 

Facilities 

60  or  more 

0 

2 

50-59 

0 

5 

40-49 

3 

7 

30-39 

6 

6 

20-29 

10 

6 

10  19 

10 

5 

0-9 

17 

1 

Task  Clusters 

The  results  of  the  factor  analysis  are  displayed  in  Appendices  C and  D,  which  lists  the  tasks,  duties, 
activities  and  work  environments  making  up  each  cluster.  Opposite  each  task  statement  is  the  factor 
loading,  a measure  which  expresses  how  important  the  task  or  duty  was  in  defining  the  factor.  A factor 
loading  can  range  up  to  a maximum  of  1.0.  A task  or  duty  had  to  exceed  .3  (an  accepted  rule-of-thumb)  in 
order  to  be  included  in  the  cluster.  An  airman  who  performed  only  those  tasks  whose  loading  is  high  on  a 
factor  had  a job  which  better  fit  the  cluster  than  did  an  airman  who  performed  only  those  tasks  whose 
loadings  were  low.  The  second  column  of  figures  in  Appendices  C and  D gives  the  proportion  of  graduates 
rated  as  having  performed  the  task.  This  proportion  was  computed  on  the  entire  sample,  not  just  on  airmen 
assigned  to  the  cluster  in  question. 

The  factor  analyses  reconfirmed  the  heterogeneity  found  in  the  occupational  survey  report.  In  the 
graduate  evaluation,  14  task  clusters  were  found  in  IM  and  8 in  MF.  Twenty-nine  clusters  were  derived  in 
the  occupational  survey.  Differences  between  the  two  surveys  were  at  least  partly  attributed  to  differences 
in  instruments  used,  populations  sampled,  and  clustering  procedures. 

In  spite  of  the  differences  between  the  two  surveys,  much  similarity  existed  between  the  two  sets  of 
clusters.  For  example,  IM  Factor  1,  Allowance  and  Authorization,  was  very  much  like  group  66  of  the 
occupational  survey  report  (OSR  printout  pp.  168-175).  The  seven  most  highly  loaded  tasks  in  Factor  1 
were  all  among  the  top  twenty  tasks  of  group  66  on  the  occupational  survey  (the  occupational  survey  tasks 
being  ranked  by  the  average  percent  of  time  spent  by  members  of  the  group  performing  the  task).  The  top 
task  was  the  same  in  both  cases,  namely,  “prepare/complete  custodian  request  receipt."  In  contrast,  task 
31,  according  to  instructor  personnel  from  the  IM  course  staff,  did  not  really  belong  in  this  cluster.  It  may 
be  seen  that  its  loading  was  much  lower  than  the  loadings  of  other  tasks  in  the  factor. 
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Considering  IM  only,  those  factors,  listed  below  were  identified  with  the  indicated  OSR  clusters: 


Graduate  Evalua- 
tion Factors 

1 

") 

3 

4 

5 

6 
7 
9 

1 I 

13 


Occupational  Sur- 
vey Cluster 
66 
131 
236 
722 
48 
55 
390 

54 

55 
577 


The  degree  of  overlap  between  corresponding  clusters  varied,  but  in  all  cases  a number  of  the  graduate 
evaluation  tasks  appeared  among  the  top  twenty  tasks  in  the  OSR. 

In  contrast,  the  tasks  in  Factor  8,  Special  Level  Detail  Maintenance,  were  not  prominent  among  the 
top  20  tasks  in  any  of  the  occupational  survey  clusters.  Likewise,  Factor  10,  Equipment  Control  (with  only 
two  tasks)  and  Factor  14,  Materiel  Control,  showed  little  correspondence  with  any  of  the  clusters  in  the 
occupational  survey.  The  tasks  of  Factor  12,  Research  Publications,  were  found  among  the  top  tasks  in 
clusters  236,  207,  and  250,  one  of  which  was  the  primary  research  cluster  in  the  occupational  survey. 
Factors  6 and  1 1 were  both  included  in  the  above  table,  but  it  should  be  noted  that  they  both  related  to 
the  same  occupational  survey  cluster,  number  55. 

The  duty  type  variable  provided  additional  means  of  interpreting  the  clusters.  It  was  of  interest  to 
note  cases  where  a duty  type  failed  to  be  associated  with  any  of  the  task  clusters  and  where  a task  cluster 
failed  to  be  associated  with  a duty  type.  For  IM,  each  of  the  duty  types  came  out  on  one  of  the  14  clusters. 
Since  the  duty  type  “other”  was  consistently  associated  with  punch  card  accounting  machine  (PCAM) 
activity,  PCAM  should  be  listed  in  future  questionnaires  as  a duty  type.  Inasmuch  as  there  were  10  duty 
types  and  14  task  clusters,  four  of  the  clusters  were  not  characterized  by  a duty  type.  Those  four  clusters 
were  1,  8,  9,  and  12.  Experts  in  the  field  may  be  able  to  provide  better  job  titles  for  these  clusters  than 
those  in  Table  2,  however,  it  was  clear  that  these  duties  represented  identifiable  task  clusters  and  should 
appear  on  future  field  questionnaires. 

For  the  MF  data,  where  only  eight  significant  task  clusters  were  found,  two  of  the  duty  types  were 
not  represented  in  the  sample  of  graduates  at  all  (inspection  and  BCE  support).  This  could  represent 
inadequate  sampling  or  nonexistence  of  such  duty  types  in  the  field.  On  the  other  hand,  two  duty  types 
(base  service  store  and  bench  stock)  nominally  existed  in  the  field,  but  there  was  no  identifiable  cluster  of 
tasks  associated  with  these  duties.  This  could  be  due  to  the  fact  that  the  appropriate  tasks  were  not 
included  on  the  questionnaire  or  that  there  was  little  or  no  commality  of  performed  tasks  from  one  duty 
station  to  another  for  these  tasks.  Also  in  the  MF  sample,  two  task  clusters  (2  and  3)  were  identified  that 
had  no  associated  duty  types.  These  were  tentatively  labeled  as  “receiving”  and  “ warehousing/ remote 
devices.”  These  duty  types  should  be  included  on  future  questionnaires.  Also,  cluster  5 was  consistently 
associated  with  D10,  which  was  “other."  This  cluster  involves  inventory  tasks  and  should  probably  be 
broken  out  as  a separate  duty  type. 
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Table  2.  IM/MF  Factors  I TatJt  Clusters) 


Factor 

Talk  Clutter? 

Inventory  Management 

i 

Allowance  and  Authon/ation 

Demand  Processing 

3 

Research  (File  Maintenance) 

4 

Repair  Cycle  Support 

5 

Individual  Equipment 

6 

Requirements  and  Requisitioning  1 

7 

Inventory 

8 

Special  Level  Detail  Record  Maintenace 

9 

Document  Control 

10 

Equipment  Control 

11 

Requirements  and  Requisitioning  II 

12 

Research  Publications 

13 

PC  AM  (Other) 

14 

Materiel  Control 

Materiel  Facilities 

1 

Warehousing  (issuing) 

2 

Receiving 

3 

Warehousing/Remote  Devices 

4 

Receiving  and  Documentation 

5 

Warehousing 

6 

Pickup  and  Delivery 

7 

War  Readiness  Materiel  (WRM) 

8 

Supply  Point 

Training  Indices 

The  ratings  for  the  tasks  in  each  of  the  task  clusters  were  averaged  across  supervisors  to  evaluate  the 
rated  proficiency  of  the  task  cluster.  Recall  that  after  the  factor  analysis,  scores  for  the  graduates  on  each 
of  the  task  clusters  were  computed.  A high  score  for  a particular  cluster  indicated  that  there  was  a high 
sinalarity  between  the  graduate’s  rated  tasks  and  the  tasks  in  the  task  cluster.  Thus,  graduates  were 
allocated  to  one  or  more  task  clusters  if  their  score  on  that  cluster  was  sufficiently  high.  Tables  3 and  4 
present  data  regarding  the  actual  ratings  within  clusters  for  1M  and  MF,  respectively.  The  column  marked 
“Graduates"  indicates  the  actual  number  of  graduates  assigned  to  the  task  cluster.  The  column  marked  N 
indicates  the  number  of  rated  tasks  for  graduates  allocated  to  a task  cluster.  A task  was  considered  rated  if 
the  rating  was  other  than  5 (not  performed).  Small  Ns  (less  than  20)  indicate  that  the  rating  data  will  have 
lov  reliability,  and  that  we  cannot  have  great  confidence  in  the  data  for  that  cluster. 

The  column  marked  “Mean  Standard”  is  the  average  STS  requirement  for  the  tasks  found  in  the 
cluster,  while  “Mean  Rating”  indicates  the  average  rating  assigned  oy  supervisors  to  the  tasks  in  the  task 
cluster.  The  Training  Index  (TI)  was  merely  the  extent  to  which  supervisor  ratings  differed  from  STS 
reuuirements.  Except  for  the  contaminating  influence  of  on-the-job  experience,  TI  might  be  regarded  as  an 
index  of  over  or  undertraining- positive  numbers  indicating  overtraining  and  negative  numbers  indicating 
une'ertraining.  In  that  sense,  all  clusters  are  overtrained  except  for  cluster  7 in  MF.  It  should  be  kept  in 
mind  that  because  a cluster  was  over-  or  undertrained,  this  did  not  imply  that  every  task  in  the  cluster  was 
over  or  undertrained,  since  TI  was  an  average  over  tasks. 

These  data  should  be  interpreted  in  light  of  the  last  two  coluntns  in  the  tables.  “Percent  Performed” 
indicates  the  average  performance  rate  of  the  tasks  within  a cluster.  That  is  to  say,  the  average  performance 
rate  for  tasks  in  a cluster  over  the  entire  sample.  “Percent  Below  Standard"  indicates  the  percent  of  tasks 
within  a cluster  which  were  rated  as  being  below  STS  requirements. 
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Table  j.  Inventory  Management  Job  Clusters 


Cluster 

Graduates 

N* 

Mean 

Standard 

Mean 

Rating 

Training 

Index® 

Percent 

Performed 

Percent 

Below 

Standard 

i 

q 

44 

2.0 

2.1 

+0.1 

2.8 

36.0 

-> 

28 

118 

1.6 

3.1 

+ 1.5 

35.0 

2.0 

3 

18 

68 

1.8 

2.7 

+0.9 

26.0 

15.0 

4 

18 

63 

2.0 

2.9 

+0.9 

12.6 

5.0 

5 

15 

55 

2.0 

1.8 

+0.8 

14.0 

18.0 

6 

20 

69 

1.83 

2.7 

+0.9 

19.8 

7.0 

7 

18 

43 

2.0 

11 

+0.2 

1 1.3 

21.0 

8 

13 

66 

1.9 

2.0 

+0.1 

10.6 

35.0 

9 

31 

120 

1.67 

3.2 

+0.5 

25.8 

1.0 

10 

1 1 

10 

1.50 

3.2 

+ 1.7 

8.0 

0.0 

1 1 

10 

47 

1.67 

3.1 

+ 1.5 

27.8 

0.0 

12 

34 

95 

2.0 

2.8 

+0.8 

27.3 

7.0 

13 

34 

102 

1.8 

2.7 

+0.9 

25.2 

10.0 

14 

16 

15 

1.33 

11 

+0.9 

16.3 

20.0 

aN=N umber  of  rated  tasks  for  graduates  assigned  to  a task  cluster. 

'’Positive  numbers  indicate  overtraining;  negative  numbers  indicate  undertraining. 


Table  4.  Materiel  Facilities  Job  Clusters 


Percent 


Cluster 

Graduates 

Na 

Mean 

Standard 

Mean 

Rating 

Training 

Index6 

Percent 

Performed 

Below 

Standard 

l 

13 

1 17 

2.0 

3.3 

+ 1.3 

43.3 

1.7 

18 

136 

2.0 

3.1 

+ 1.1 

41.9 

4.4 

3 

9 

89 

2.0 

2.3 

+0.3 

25.0 

24.7 

4 

6 

20 

2.0 

3.0 

+ 1.0 

28.1 

0.0 

5 

5 

33 

2.0 

3.1 

+ 1.1 

29.9 

9.1 

6 

9 

25 

2.0 

2.8 

+0.8 

60.6 

12.0 

7 

6 

40 

win 

1.4 

-0.6 

23.2 

60.0 

8 

5 

7 

ill 

3.3 

+ 1.3 

32.6 

0.0 

aN=Number  of  rated  tasks  for  graduates  assigned  to  a task  cluster. 

"Positive  numbers  indicate  overtraining:  negative  numbers  indicate  undertraining. 


The  relationship  between  T1  and  “Percent  Performed”  is  plotted  in  Figure  2.  The  plot  has  been 
divided  into  three  non-overlapping  areas  labeled  “overtraining,”  “undertraining,”  and  “cost-effective 
training.”  The  manner  in  which  the  three  areas  were  formed  was  arbitrary  in  the  sense  that  particular 
partitioning  is  an  expression  of  system  philosophy  about  the  goals  of  resident  training.  For  example,  the 
overtraining  area  included  any  task  cluster  which  was  performed  by  less  than  10%  of  the  sample  and  any 
cluster  with  a TI  greater  than  1 .0.  Perhaps  a more  reasonable  partition  would  include  any  cluster  performed 
by  less  than  20%’  of  the  sample.  As  can  be  seen,  the  cost-effective  training  zone  is  a triangle.  The 
hypotenuse  of  the  triangle  is  an  expression  of  the  philosophy  that  relative  overtraining  could  be  tolerated 
for  tasks  performed  with  high  frequency,  while  relative  undertraining  could  be  tolerated  for  tasks 
performed  with  low  frequency.  Of  course,  these  are  broad  generalizations  which  may  not  hold  for  specific 
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Figure  2.  Relation  between  training  index  and  percent  performing. 


1 


situations.  For  example,  tasks  required  in  the  event  of  a breach  in  safety  will  be  performed  with  very  low 
frequency  by  all;  however,  one  would  certainly  argue  that  it  is  essential  that  everybody  be  highly  trained  in 
these  tasks.  These  considerations  imply  that  there  are  other  dimensions  to  consider  when  dealing  with 
cost-effective  training.  The  above  example  indicates  that  a dimension  such  as  consequences  of 
first-perfomiance-failure  may  override  other  considerations.  At  any  rate,  it  is  clear  an  operationalized 
partition  has  important  implications  for  cost-effective  technical  training. 

Based  on  the  example  partition  in  Figure  2,  there  were  a number  of  task  clusters  which  seemed  to  be 
overtrained:  clusters  1,2,  10,  and  11  in  IM  and  I,  2,  5,  and  8 in  MF.  On  the  other  side,  cluster  8 in  I M and 
3,  6,  and  7 in  MF  seemed  to  be  undertrained.  There  were  a total  of  ten  clusters  falling  into  the 
cost-effective  zone.  If  the  overtraining  zone  included  tasks  performed  by  less  than  20%  of  the  sample,  there 
would  be  only  four  task  clusters  in  the  cost-effective  zone. 

An  interesting  point  was  that  the  so-called  cost-effective  zone  was  a relatively  small  portion  of  the 
entire  area.  Thus,  accomplishing  cost-effective  training,  at  least  for  these  two  courses,  represents  a 
considerable  training  challenge. 

Unclassifiable  Graduates 

It  was  pointed  out  earlier  that  the  sampling  procedure  had  an  important  influence  on  the  task  clusters 
which  were  identified.  Most  of  the  clusters  found  in  each  sample  paralleled  those  found  in  the  OSR. 
Somewhat  more  in-depth  analysis  was  available  by  examining  the  graduates  who  were  not  classifiable  into 
any  of  the  clusters. 

Some  graduates  were  not  classified  into  any  of  the  job  clusters.  An  unclassifiable  graduate  resulted 
when  scores  on  all  clusters  were  low;  essentially,  these  graduates  performed  clusters  of  tasks  which  were 
unlike  any  other  cluster  in  the  sample.  Tire  percentage  of  such  graduates  was  rather  low  in  IM  (9.8%); 
however,  about  25%  of  the  MF  graduates  were  unclassifiable. 

To  some  extent,  these  graduates  could  be  airmen  who  were  assigned  outside  base  supply,  and  who  for 
that  reason  had  somewhat  idiosyncratic  jobs.  If  the  jobs  were  not  idiosyncratic,  then  questionnaires  may 
have  been  deficient  in  not  representing  the  range  of  tasks  to  which  the  3-level  graduate  was  assigned.  This 
situation  could  be  prevented  if,  before  the  survey,  experts  in  the  specialties  compared  the  graduate 
evaluation  questionnaires  with  previous  OSRs. 


IV.  DISCUSSION  AND  RECOMMENDATIONS 


Optimal  Evaluation  Instrument 

Ideally,  an  evaluation  instrument  should  allow  the  evaluator  to  make  meaningful  recommendations 
concerning  task  clusters  as  well  as  individual  tasks.  These  recommendations  should  allow  training  personnel 
to  evaluate  instruction  in  the  light  of  operationally  defined  job  types.  That  is,  this  type  of  analysis  gives  the 
training  personnel  an  idea  of  the  tasks  which  constitute  a job  and  the  relative  rated  proficiency  with  which 
the  job  is  performed.  This  type  of  analysis  also  has  the  advantage  of  removing  tasks  irrelevant  to  the  job  and 
training  students  only  on  required  tasks. 

The  evaluation  instrument  should  emphasize  how  tasks  are  clustered  together  by  duty  types.  By  this, 
we  mean  that  during  the  development  of  the  evaluation  instrument,  the  evaluator  should  attempt  to  ensure 
that  jobs  which  exist  in  the  field  are  represented  in  the  questionnaire  by  appropriate  tasks  and  duty  types. 
Earlier  occupational  survey  reports  can  be  used  to  suggest  clusters  that  should  be  represented.  To  provide 
better  coverage  of  the  various  clusters,  the  questions  on  the  graduate  evaluation  questionnaire  concerned 
with  duty  assignment,  work  environment  and  activities  should  be  coordinated  to  give  a progressively  more 
refined  description  of  the  graduate's  location  in  the  organization  and  his  major  responsibilities.  For 
example,  a questionnaire  might  include  as  duty  types  all  the  sections  within  an  organization;  the  work 
environment  variables  might  include  all  the  units  within  sections;  and  the  activities  might  be  major 
functions  performed  within  the  units.  In  this  way,  the  evaluator  can  denote  each  task  cluster  by  very 
specific  descriptions. 

The  optimal  evaluation  instrument  should  also  allow  the  evaluator  to  make  specific  recommendations 
about  which  tasks  are  best  taught  together.  For  example,  the  analysis  of  the  IM  and  MF  surveys  suggests 
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that  tasks  associated  with  specific  task  clusters  might  be  taught  together  as  a unit  of  instruction.  These 
recommendations  are  based  on  objective  evidence  gathered  from  the  field.  That  is,  field  supervisors  have 
reported  that  a specific  duty  tends  to  be  made  up  of  a cohesive  cluster  of  tasks. 

Of  course,  the  instrument  should  allow  the  evahi-tor  to  draw  conclusions  about  the  adequacy  of 
training  received  in  the  course.  Our  analysis  allowed  the  comparison  of  percent  performed,  training  index, 
and  percent  below  standard.  These  measures  have  implications  for  course  design  (i.e.,  amount  of  training 
required)  as  well  as  providing  a measure  of  training  adequacy.  For  example,  if  a small  percentage  of  the 
population  performs  a task  cluster  and  the  average  performance  is  rated  well  above  standard,  tills  suggests 
that  perhaps  the  course  is  overtraining  individuals  performing  that  job.  Likewise,  if  the  training  index  is 
negative  or  only  slightly  positive,  a high  percentage  of  the  population  performing  the  job,  and  a high 
percentage  (20%)  rated  as  below  standard,  this  suggests  that  training  for  that  specific  job  is  inadequate. 

It  should  be  noted  that  field  evaluations  for  a particular  course  arc  conducted  approximately  every 
two  years.  This  has  the  effect  of  leaving  the  feedback  loop  open  (see  Goal  of  Effective  Field  Evaluation). 
After  a course  is  evaluated,  recommendations  for  changes  are  made;  however,  the  impact  of  the  changes  is 
not  measured  again  for  two  years.  This  lapse  has  the  effect  of  blurring  the  entire  evaluation  process.  The 
methodology  utilized  here  employed  only  three  major  variables  plus  a set  of  task  descriptors.  A short-form 
questionnaire  could  be  used  as  a check  on  any  course  modifications  instituted  as  a result  of  the  original 
survey.  This  in  effect  would  close  the  feedback  loop  and  furnish  timely,  pertinent  data  to  instructional 
personnel. 

Modification  of  the  Field  Evaluation  System 

The  results  of  this  study  suggest  some  modifications  to  the  field  evaluation  system,  especially  in  the 
areas  of  questionnaire  design,  data  analysis,  and  interpretation  of  results.  The  cornerstone  of  any  survey 
technique  is  the  instrument  used  to  gather  the  data.  Our  results  suggest  that  the  instruments  being  used  in 
the  present  field  evaluation  system  could  be  improved  in  several  ways.  First,  the  tasks  which  are  included  in 
the  instrument  should  be  carefully  selected  so  that  they  are  representative  of  those  tasks  which  are 
performed  frequently  by  job  incumbents.  Although  the  STS  provides  a criteria  for  performance,  one  cannot 
rely  solely  on  the  STS  for  the  selection  and  definition  of  tasks.  In  developing  a field  evaluation 
questionnaire,  previous  OSRs  and  evaluation  reports  should  be  carefully  studied  and  the  findings  discussed 
with  experienced  personnel  from  the  fields  being  evaluated.  The  results  of  this  exercise  may  reveal  certain 
inadequacies  and  ambiguities  in  the  STS.  which  need  to  be  taken  into  account  during  the  survey,  and  which 
should  lead  to  revision  of  the  STS.  Second,  the  duty  descriptors  should  be  selected  so  that  they  represent 
the  known  jobs  in  the  field.  Finally,  the  work  environment  and  activity  variables  should  be  selected  so  that 
they  provide  a molecular  description  of  the  job.  In  this  way,  a task  cluster  can  be  defined  in  very  molar 
(i.e.,  duty)  or  molecular  (i.e.,  work  environment)  terms. 

The  types  of  data  analysis  performed  in  this  study  also  allow  the  evaluator  to  make  recommendations 
to  the  course  concerning  areas  of  possible  undertraining  or  overtraining,  and  about  the  organization  of 
training  material.  Furthermore,  there  are  strong  implications  about  how  training  goals  should  be  modified. 
For  example,  our  analysis  of  the  IM  course  suggests  that  a combination  of  resident  training  and  OJT  leads 
to  substantial  overtraining  on  cluster  2,  (Demand  Processing)  and  cluster  11,  (Requirements  and 
Requisitioning  II).  Tliis  type  of  empirical  data  is  what  is  needed  by  training  personnel  when  making 
decisions  about  increasing  or  decreasing  training  in  a particular  area.  The  point,  really,  is  that  for  a field 
evaluation  process  to  be  viable,  the  evaluator  must  make  concrete  recommendations  for  curriculum  change, 
rather  than  merely  presenting  data  for  a list  of  tasks. 

Throughout  the  conduct  of  the  present  effort,  the  investigators  were  continually  struck  by  the 
consistency  with  which  supervisors  responded  to  questions  about  which  tasks  were  being  performed.  The 
task  clusters  were  crisp  and  clearly  defined.  This  indicates  that  our  field  personnel  arc , by  and  large,  making 
an  effort  to  provide  the  best  possible  input  for  course  evaluation.  Because  of  this,  we  should  make  every 
effort  to  optimize  the  utility  of  field  evaluation. 

Recommendations 

In  summary,  we  suggest  that  the  field  evaluation  system  be  modified  in  several  ways.  First,  more 
flexibility  should  be  introduced  into  the  data  analysis  capability  so  that  standard  statistical  techniques,  such 
as  factor  analysis  and  regression  analysis  can  be  used.  The  present  analysis  system  (ATCM  52  334) 
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essentially  is  limited  to  summarizing  percentages  by  task,  and  does  not  have  the  capability  for  in-depth 
analysis  such  as  was  done  here.  Second,  recommendations  to  the  course  should  be  meaningful  and  specific. 
For  example,  the  problems  of  undertraining  and  overtraining  should  be  addressed  along  with  suggested 
reorganization  of  training  materials,  if  appropriate.  And  finally,  all  recommendations  should  be  made  by 
task  clusters,  as  well  as  specific  tasks.  This  approach  reveals  which  jobs  need  more  or  less  training  rather 
than  individual  tasks  which  in-  and  of-themselves  may  not  be  critical  in  the  successful  performance  of  a job. 
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APPENDIX  A:  INVENTORY  MANAGEMENT  INSTRUMENT 


Duties 

D1  Equipment  Control 

D2  Individual  Equipment 

D3  Requirements  and  Requisitioning 

D4  Research 

D5  Demand  Processing 

D6  Inventory 

D7  Stock  Control 

D8  Repair  Cycle  Support 

D9  Material  Control 

DIO  Other 

Work 

W1  Punch  Card  Accounting  Machines  (PCAM) 

W2  Requisitioning  of  Property 

W3  Materiel  Control  - Records  Validation 

W4  Stock  Control  Clerk 

WS  Operating  Remote  Devices 

W6  Delivering  Supplies  and  Equipment 

W7  Researching  Publications  and  Files 

W8  Maintaining  Records,  Files  and  Publications 

W9  Inventorying  Supplies  and  Equipment 

W10  Other 

Activities 

A1  Set  Up  Work  Process  Files 

A2  Establishment,  Change,  and  Deletion  of  A Special  Level  Detail  Record 
A3  Perform  Final  Quality  Control  of  Source  Documents 

A4  Reporting  of  Excess  Material 

A5  Log  in  AF  Forms  601b  and  Assign  to  Each  an  EMO  Control  Number 
A6  Identify  Delinquent  Documents 

A7  Assignment  of  Equipment  Authorization  Inventory  Data  (EAID) 
Authorized  In-Use  Detail  Document  Number 
A8  Annotation  of  AF  Forms  601b  with  Date  and  Transaction  Serial  Number 
A9  Inventory  Supplies  and  Equipment 

A10  Prepare  and  Input  Warehouse  Location  Data 
A1 1 Initiate  Follow-up  Action  on  Requisitions 

A12  Correct  or  Annotate  Issue  or  Shipping  Documents 
A13  Deliver  Bench  Stock  Items 

A14  Prepare  AF  Forms  2005,  Issue  Request  for  Routine  and  Priority  Issues 

A1 5 Drive  Car  or  Pickup  Truck 

A16  Maintain  Publications 

A17  Research  Publications 

A18  Maintain  Records  and  Files 

A19  Research  Files 

A20  Update  Stock  Number  Directory 

A21  Prepare  and  Annotate  Turn-In  Documents 

A22  Other 

Tasks 

TI  Conduct  sample/cycle,  special  and  EAID  inventories 
T2  Correct  inventory  discrepancies 
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T3  Research  standard  publications  to  determine  policy 
T4  Prepare  inputs  to  load,  change,  or  delete  item  records 

T5  Prepare  inputs  to  load,  change,  or  delete  interchangeability  and 

substition  group  records 

T6  Maintain  work  listings  and  external  files 

T7  Prepare  non-federal  stock  number  control  forms 

T8  Maintain  computer  input  suspense  tiles 

Tl)  Research  supply  publications  for  supply  transaction  data 

T10  Research  technical  publications  for  supply  transaction  data 

Tl  1 Distribute  automatic  data  processing  equipment  products 

T12  Maintain  document  and  support  files 

T13  Identify  delinquent  documents 

T14  Operate  remote  keyboard  printer 

T15  Operate  keypunch 

TI6  Edit  File  Maintenance  Transactions 

T 1 7 Prepare  inputs  to  load,  change,  or  delete  Exception  Codes 

T18  Prepare  inputs  to  load,  change,  or  delete  Special  Levels 

T 10  Complete  forms  and  request  approval  for  special  levels 

T20  Maintain  special  level  justification  data 

T21  Prepare  forced  due-out  release  inputs 

T22  Revalidate  due-outs  and  due-ins 

T23  Prepare  documentation  and  inputs  for  off-line  shipments 

T24  Prepare  documentation  and  inputs  for  off-line  transfers  to 

Defense  Property  Disposal  Activity 
T25  Prepare  documentation  and  inputs  for  off-line  requisitions 
T26  Prepare  issue  inputs  for  computer  processing 
T27  Prepare  turn-in  inputs  for  computer  processing 

T28  Maintain  Control  Register 

T29  Process  management  notices,  reject  and  inquiries 

T30  Process  post  transactions 

T31  Establish  and  maintain  authorization  lists  and  master  bench  stock 
cards 

T32  Prepare  supply  point  input  transactions 
T33  Maintain  the  Due-In  from  Maintenance  (D1FM)  suspense  file 
T34  Prepare  inputs  to  update  DIFM  detail  records 
T35  Prepare  turn-in  documents  for  DIFM  items 

T36  Prcpare/complete  custodian  request-receipts 

T37  Research  allowance  documents  to  determine  equipment  allowances 

T38  Prepare  inputs  to  load,  change,  or  delete  Equipment  Authorization 

Inventory  Data  (EAID)  in-use  detail  records 
T39  Prepare  inputs  to  record  transfers  of  equipment  between  custody 
accounts 

T40  Screen,  update,  and  file  allowance  source  code  listings 

T41  Screen,  update,  and  file  Custodian  Authorization-Custody  Receipt 

Listings  and  related  products 

T42  Screen,  update,  and  file  allowance  and  authorization  documents 
T43  Document  and  issue  and  turn-in  of  personal  retention  items 

T44  Document  and  issue  and  turn-in  of  organization  equipment  items 

T45  Document  and  issue  and  turn-in  of  tools  and  tool  kits 

T46  Document  and  issue  and  tum-in  of  mobility  items  and  kits 
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Duties 

D1  Receiving 

D2  Inspection 

D3  Warehousing 

D4  Pickup  and  Delivery 

D5  Base  Service  Store 

D6  WRM 

D7  Supply  Point 

D8  BCE  Support 

D9  Bench  Stock 

DIO  Other 

Work 

W1  Receiving  Property 

W2  Inspecting,  Identifying  and  Classifying  Property 
W 3 Sto  ring  Supplies  and  Eq  ui  pme  n t 

W4  Issuing,  Shipping  and  Transferring  Supplies  and  Equipment 

W5  Operating  Remote  Devices 

W6  Delivering  Supplies  and  Equipment 

W7  Researching  Publications  and  Files 

W8  Maintaining  Records,  Files,  and  Publications 

W9  Inventorying  Supplies  and  Equipment 

W10  Other 

Activities 

A1  Offload  supplies  and  equipment 

A2  Compare  physical  count  with  information  contained  in  receiving 
document,  tags,  labels  and  markings 
A3  Prepare  extract  receiving  documents 

A4  Move  property  to  areas  such  as  delivery  or  storage 
A5  Input  receiving  information  to  computer 

A6  Place  property  in  warehouse  bins,  racks,  or  bays 
A7  Move  property  to  pick-up  and  delivery  units 
A8  Locate  items  to  be  issued,  shipped,  or  transferred 
A9  Inventory  supplies  and  equipment 

A10  Prepare  and  input  warehouse  location  data 
A1 1 Prepare  or  correct  bin  labels 

A1 2 Correct  or  annotate  issue  or  shipping  documents 
AI3  Deliver  property  items 

A14  Operate  materiel  handling  equipment 

A15  Drive  car  or  pickup  truck 

A16  Maintain  publications 

A17  Research  publications 

A18  Maintain  records  and  files 

A19  Research  files 

A20  Update  stock  number  directory 

A21  Prepare  and  annotate  turn-in  documents 

A22  Other 

Tasks 

T1  Perform  remote  device  operator  maintenance 

T2  Gear  rejects 
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T3  Verify  operating  procedures 

T4  Prepare  labels  and  tags  for  property 
T5  Prepare/ process  stock  change  documents 

T6  Prepare  property  for  inchecking 

T7  Compare  information  on  tags  and  labels  against  documentation 

T8  Prepare  extract  documents 

T9  File,  update,  and  input  receipt  due-in  cards 

T10  Enter  information  on  receiving  documents 

T1 1 Process  management  notices 

T12  Move  property  to  holding  area 

T13  Move  property  to  delivery  area 

T14  Move  property  to  storage  area 

T15  Prepare  turn-in  documents 

T16  Operate  remote  keyboard  printer 

T17  Operate  remote  card  reader 

T18  Place  location  symbols  on  storage  facilities 

T19  Prepare  and  correct  bin  labels 

T20  Update  stock  number  directory 

T21  Prepare  and  input  warehouse  location  data 

T22  Process  output  documents  and  listings 

T23  Locate  and  select  property  as  indicated  on  documents 

T24  Complete  document  entries 

T25  Move  property  to  delivery  point 

T26  Prepare  warehouse  for  inventory 

T27  Make  emergency  issues 

T28  Process  inquiries 

T29  Apply  proper  methods  for  handling  property 
T30  Obtain  authorized  representative  signatures 

T31  Maintain  warehouse  document  file 

T32  Distribute  documents 


API'I  SDIX  C:  INVENTORY  MANAGEMENT  FACTORS 


Proportion 


Task  Nr 

Task  Statement 

Loading 

Performing' 

Factor  1 - 

Allowance  and  Authorization 

T36* 

Prepare/ complete  custodian  request  receipts 

.88 

.04 

T42 

Screen,  update  and  Tile  allowance  and 

authorization  documents 

.85 

.03 

T41 

Screen,  update  and  file  custodian-authorization 
custody  receipt  listings  and  related  products 

.82 

.05 

T39 

Prepare  inputs  to  record  transfers  of  equipment 

between  custody  accounts 

.81 

.04 

T37 

Research  allowance  documents  to  determine 

equipment  allowances 

.76 

.04 

T40 

Screen,  update  and  file  allowance  source 

code  listings 

.73 

.02 

T38 

Prepare  inputs  to  load,  change  or  delete 

Equipment  Authorization  Inventory 
Data  (EA1D)  in-use  detail  records 

.65 

.04 

A07** 

Assignment  of  EA1D  authorized  in-use  detail 

document  number 

.54 

.01 

T31 

Establish  and  maintain  authorization  lists 

and  master  bench  stock  cards 

.35 

.05 

Factor  2 - 

Demand  Processing 

D5(" 

Demand  processing 

.83 

.15 

T28 

Maintain  control  register 

.82 

.31 

T30 

Process  post  transactions 

.81 

.29 

T26 

Prepare  issue  inputs  for  computer  processing 

.79 

.41 

T29 

Process  management  notices,  reject  and 

inquiries 

.72 

.49 

A14 

Prepare  AF  Forms  2005,  Issue  request  for 

routine  and  priority  issues 

.65 

.34 

W5# 

Operating  remote  devices 

.46 

.10 

T27 

Prepare  turn-in  inputs  for  computer  processing 

.38 

.26 

Factor  3 - 

Research  (File  Maintenance) 

D4 

Research 

.81 

.11 

T4 

Prepare  inputs  to  load,  change  or  delete 

item  records 

.78 

.19 

T5 

Prepare  inputs  to  load,  change  or  delete 

interchangeability  and  substitution 
group  records 

.78 

.09 

T7 

Prepare  non-federal  stock  number  control 

forms 

.67 

.14 

W7 

Researching  publications  and  files 

.63 

.09 

A20 

Update  stock  number  directory 

.51 

.13 

T14 

Operate  remote  keyboard  printer 

.34 

.49 

T6 

Maintain  work  listings  and  external  files 

.29 

.39 

Factor  4 - 

Repair  Cycle  Support 

T34 

Prepare  inputs  to  update  DIFM  detail  records 

.92 

.10 

T35 

Prepare  tum-in  documents  for  DIFM  items 

.89 

.11 
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Proportion 

Task  Nr 

Task  Statement 

Loading 

Performing* 

Factor  4 

— Repair  Cycle  Support  (Continued) 

T33 

Maintain  the  DIFM  suspense  tile 

.88 

.1 1 

1)8 

Repair  cycle  support 

.85 

.07 

T32 

Prepare  supply  point  input  transactions 

.70 

.06 

A21 

Prepare  and  annotate  turn-in  documents 

.48 

22 

T27 

Prepare  turn-in  inputs  for  computer  processing 

.31 

.26 

Factor  5 

Individual  Equipment 

T43 

Document  and  issue  and  turn-in  of  personal 

retention  items 

.89 

.07 

D2 

Individual  equipment 

.85 

.04 

T44 

Document  and  issue  and  turn-in  of  organization 
equipment  items 

.85 

.08 

A10 

Prepare  and  input  warehouse  location  data 

.51 

.07 

T46 

Document  and  issue  and  turn-in  of  mobility 

items  and  kits 

.48 

.04 

T45 

Document  and  issue  and  turn-in  of  tools  and 

tool  kits 

.40 

.06 

A21 

Prepare  and  annotate  turn-in  documents 

.37 

.22 

T27 

Prepare  turn-in  inputs  for  computer  processing 

.32 

.26 

TI7 

Prepare  inputs  to  load,  change  or  delete 

Exception  Codes 

.30 

.15 

Factor  6 

— Requirements  and  Requisitioning  1 

D3 

Requirements  and  requisitioning 

.85 

.14 

T22 

Revalidate  due-outs  and  due-ins 

.70 

.29 

T23 

Prepare  documentation  and  inputs  for  offline 

shipments 

.60 

.20 

T25 

Prepare  documentation  and  inputs  for  offline 

requisitions 

.56 

.26 

T21 

Prepare  forced  due-out  release  inputs 

.56 

.25 

W4 

Stock  control  clerk 

.54 

.13 

Al  1 

Initiate  follow-up  action  on  requisitions 

.46 

.31 

T24 

Prepare  documentation  and  inputs  for  offline 

transfers  to  Defense  Property  Disposal 
Activity 

.42 

.12 

T19 

Complete  forms  and  request  approval  for 

special  levels 

.31 

.07 

W2 

Requisitioning  of  property 

.30 

.1 1 

Factor  7 

- Inventory 

T2 

Correct  inventory  discrepancies 

.89 

.14 

D6 

Inventory 

.85 

.06 

Tl 

Conduct  sample/cycle,  special  and  EAID 

inventories 

.82 

.17 

W9 

Inventorying  supplies  and  equipment 

.76 

.08 

A9 

Inventory  supplies  and  equipment 

.54 

.20 

T38 

Prepare  inputs  to  load,  change  or  delete 

Equipment  Authorization  Inventory  Data 
(EAID)  in-use  detail  records 

.38 

.04 

24 
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Task  Nr 

Task  Statement 

Loading 

Proportion 

Performing* 

Factor  8 - 

Special  Level  Detail  Record  Maintenance 

T20 

Maintain  special  level  justification  data 

.80 

.08 

T18 

Prepare  inputs  to  load,  change  or  delete 

special  levels 

.75 

.07 

T19 

Complete  forms  and  request  approval  for 

special  levels 

.73 

.07 

A2 

Establishment,  change  and  deletion  of  a 

special  level  detail  record 

.45 

.06 

Til 

Distribute  automatic  data  processing  equipment 

products 

.37 

.21 

T45 

Document  and  issue  and  tum-in  of  tools 

and  tool  kits 

.37 

.06 

T31 

Establish  and  maintain  authorization  lists 

and  master  bench  stock  cards 

.37 

.05 

TI7 

Prepare  inputs  to  load,  change  or  delete 

Exception  Codes 

.35 

.15 

T24 

Prepare  documentation  and  inputs  for  offline 

transfers  to  Defense  Property 

Disposal  Activity 

.30 

.12 

T16 

Edit  File  Maintenance  Transactions 

.29 

.10 

Factor  9 - 

Document  Control 

T12 

Maintain  document  and  support  files 

.77 

.30 

T8 

Maintain  computer  input  suspense  files 

.75 

.30 

T6 

Maintain  work  listings  and  external  files 

.73 

.39 

T13 

Identify  delinquent  documents 

.42 

.25 

T16 

Edit  file  maintenance  transactions 

.36 

.10 

. A18 

Maintain  records  and  files 

.34 

.52 

Til 

Distribute  ADP  equipment  products 

.34 

.21 

Factor  10 

- Equipment  Control 

Dl 

Equipment  control 

.88 

.05 

T24 

Prepare  documentation  and  inputs  for 

off-line  transfers  to  Defense  Property 

Disposal  Activity 

.56 

.12 

T46 

Document  and  issue  and  tum-in  of  mobility 

items  and  kits 

.37 

.04 

Factor  1 1 

- Requirements  and  Requisitioning  II 

D7 

Stock  control 

.84 

.05 

T25 

Prepare  documentation  and  inputs  for 

off-line  requisitions 

.60 

.26 

T2I 

Prepare  forced  due-out  release  inputs 

.55 

.25 

T23 

Prepare  documentation  and  inputs  for 

off-line  shipments 

.40 

.20 

T27 

Prepare  tum-in  inputs  for  computer 

processing 

.36 

.26 

T26 

Prepare  issue  inputs  for  computer  processing 

.35 

.41 

All 

Initiate  follow-up  action  on  requisitions 

.34 

.31 

T22 

Revalidate  due-outs  and  due-ins 

.33 

.29 
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Task  Nr 

Task  Statement 

Loading 

Proportion 

Performing* ** 

Factor  12  - 
T9 

Research  Publications 

Research  supply  publications  for  supply 
transaction  data 

.83 

.37 

T3 

Research  standard  publications  to 
determine  policy 

.70 

.23 

T10 

Research  technical  publications  for 
supply  transaction  data 

.69 

.30 

W7 

Research  publications  and  files 

.38 

.09 

T4 

Prepare  inputs  to  load,  change  or  delete 
item  records 

.32 

.19 

Factor  13  - 
T15 

Punched  Card  Accounting  Machines  (PCAM) 
Operate  keypunch 

.80 

.26 

T13 

Identify  delinquent  documents 

.55 

.25 

T14 

Operate  remote  keyboard  printer 

.40 

.49 

DIO 

Other 

.40 

.22 

T31 

Establish  and  maintain  authorization  lists 
and  master  bench  stock  cards 

.35 

.05 

TU 

Distribute  ADP  equipment  products 

.32 

.21 

W1 

Punch  card  accounting  machines 

.31 

.06 

Factor  14  - 

D9 

Material  Control 

Material  control 

.82 

.10 

W3 

Material  control  - records  validation 

.70 

.10 

T45 

Document  and  issue  and  turn  in  of  tools 
and  tool  kits 

.44 

.06 

T22 

Revalidate  due-outs  and  due-ins 

.38 

.29 

T7 

Prepare  non-federal  stock  number  control 
forms 

.38 

.14 

All 

Initiate  follow-up  action  on  requisitions 

.37 

.31 

A5 

Log  in  AF  Forms  601b  and  assign  to  each 
an  EMO  control  number 

.36 

.04 

A8 

Annotation  of  AF  Forms  601b  with  date 
and  transaction  serial  number 

.32 

.01 

A2 

Establishment,  change  and  deletion  of  a 
special  level  detail  record 

.30 

.06 

*T  Job  Task 

**A  Job  Activity 

f"D  Duty  Assignment 
aW  Type  of  Work  Environment 


APPENDIX  D MATERIEL  FACILITIES  FACTORS 


Proportion 


Task  Nr 

Task  Statement 

Loading 

Performing* 

Factor  1 

- Warehousing 

T23* 

Locate  and  select  property  as  indicated 

on  documents 

.81 

.57 

T24 

Complete  document  entries 

.74 

.24 

D3** 

Warehousing 

.72 

.30 

T2i 

Prepare  and  input  warehouse  location 

.71 

38 

T25 

Move  property  to  delivery  point 

.70 

.62 

T19 

Prepare  and  correct  bin  labels 

.65 

.52 

T20 

Update  stock  number  directory 

.57 

.35 

T18 

Place  location  symbols  on  storage  facilities 

.56 

.39 

T31 

Maintain  warehouse  document  file 

.53 

.28 

A8f« 

Locate  items  to  be  shipped,  issued,  or 

transferred 

.49 

.51 

A1 1 

Prepare  or  correct  bin  labels 

.48 

.39 

A6 

Place  property  in  warehouse  bins,  racks,  or 

bays 

.46 

.52 

J'JT 

Process  output  documents  and  listings 

.45 

.32 

A10 

Prepare  and  input  warehouse  location  data 

.44 

.23 

A18 

Maintain  records  and  files 

.39 

.25 

A20 

Update  stock  number  directory 

.37 

.22 

W3rr 

Storing  supplies  and  equipment 

.36 

.17 

T32 

Distribute  documents 

.35 

.49 

T28 

Process  inquiries 

.33 

.30 

W4 

Issuing,  shipping,  and  transferring  supplies 

and  equipment 

.31 

.20 

Factor  2 

— Receiving 

T13 

Move  property  to  delivery  area 

.89 

.46 

T14 

Move  property  to  storage  area 

.86 

.46 

T7 

Compare  information  on  tags  and  labels  against 

documentation 

.86 

.54 

T12 

Move  property  to  holding  area 

.81 

.46 

T10 

Enter  information  on  receiving  documents 

.56 

.41 

T15 

Prepare  turn-in  documents 

.56 

.36 

TU 

Process  management  notices 

.52 

.20 

T6 

Prepare  property  for  inchecking 

.50 

.41 

T8 

Prepare  extract  documents 

.49 

.26 

T4 

Prepare  labels  and  tags  for  property 

.40 

.22 

A4 

Move  property  to  areas  such  as  delivery 

or  storage 

.38 

.52 

T29 

Apply  proper  methods  for  handling  property 

.30 

.83 

Factor  3 

- Remote  Devices 

T16 

Operate  remote  keyboard  printer 

.75 

.24 

T28 

Process  inquiries 

.66 

.30 

Tl 

Perform  remote  device  operator  maintenance 

.63 

.19 

T31 

Maintain  warehouse  document  file 

.62 

.28 

T17 

Operate  remote  card  reader 

.61 

.16 

T3 

Verify  operating  procedures 

.60 

.17 

27 


Task  Nr Task  Statement 

Factor  3 - Remote  Devices  (Continued) 

T32  Distribute  documents 

T22  Process  output  documents  and  listings 

W5  Operating  remote  devices 

T2  Clear  rejects 

T5  Prepare/process  stock  change  documents 

T20  Update  stock  number  directory 

T21  Prepare  and  input  warehouse  location  data 

A20  Update  stock  number  directory 

Til  Process  management  notices 

T9  File,  update,  and  input  receipt  due-in  cards 

Factor  4 - Receiving  and  Documentation 

T10  Enter  information  on  receiving  documents 

T6  Prepare  property  for  inchecking 

T8  Prepare  extract  documents 

A3  Prepare  extract  receiving  documents 

T9  File,  update,  and  input  receipt  due-in  cards 

D1  Receiving 

T17  Operate  remote  card  reader 

Factor  5 — Inventory 
DIO  Other 

T26  Prepare  warehouse  for  inventory 

T27  Make  emergency  issues 

T2  Clear  rejects 

T3  Verify  operating  procedures 

T1  Perform  remote  device  operator  maintenance 

T20  Update  stock  number  directory 

T 1 1 Process  management  notices 

W9  Inventorying  supplies  and  equipment 

T22  Process  output  documents  and  listings 

T18  Place  location  symbols  on  storage  facilities 

T24  Complete  document  entries 

Factor  6 — Pickup  and  Delivery 
D4  Pickup  and  delivery 

T30  Obtain  authorized  representative  signatures 

A13  Deliver  property  items 

T32  Distribute  documents 

W6  Delivering  supplies  and  equipment 

T29  Apply  proper  methods  for  handling  property 

A14  Operate  materiel  handling  equipment 

A15  Drive  car  or  pickup  truck 

A1  Offload  supplies  and  equipment 

Factor  7 - War  Readiness  Materiel  (WRM) 

D6  WRM 

T4  Prepare  labels  and  tags  for  property 


Loading 


.51 

.48 

.46 

.46 

.42 

.41 

.39 

.39 

.38 

.35 


.66 

.62 

.60 

.53 

.50 

.48 

.36 


.84 

.74 

.56 

.48 

.46 

.40 

.39 

.39 

.37 

.38 

.32 

.31 


.81 

.69 

.59 

.58 

.58 

.50 

.43 

.40 

.33 


.79 

.60 


28 


Proportion 

Performing* 


49 

32 

10 

17 

07 

35 

38 

n 

20 

.17 


41 

41 

26 

22 

17 

25 

16 


.06 

.25 

.41 

.19 

.17 

.19 

.35 

.20 

.03 

.32 

.39 

.54 


.14 

.50 

.33 

.49 

.22 

.83 

.46 

.43 

.52 


.03 

.22 


Task  Nr 

Task  Statement 

Loading 

Proportion 

Performing* 

Factor  7 - 

Ware  Readiness  Materiel  ( WRM)  (Continued) 

T5 

Prepare/process  stock  change  documents 

.60 

.07 

T9 

F ilc,  update,  and  input  receipt  due-in  cards 

.51 

.17 

T17 

Operate  remote  card  reader 

.47 

.16 

A9 

Inventorying  supplies  and  equipment 

.39 

.35 

Til 

Process  management  notices 

.34 

.20 

T19 

Prepare  and  correct  bin  labels 

.34 

.52 

T18 

Place  location  symbols  on  storage  facilities 

.33 

.39 

W9 

Inventorying  supplies  and  equipment 

.33 

.16 

T3 

Verify  operating  procedures 

.32 

.17 

Factor  8 - 

Supply  Point 

D7 

Supply  point 

.80 

.01 

\V2 

Inspecting,  identifying  and  classifying 

property 

.80 

.01 

A6 

Place  property  in  warehouse  bins,  racks,  or  bays 

.40 

.52 

AK 

Locate  items  to  be  issued,  shipped,  or 

transferred 

.38 

.51 

T26 

Prepare  warehouse  for  inventory 

.37 

.25 

T27 

Make  emergency  issues 

.32 

.41 

*T  Job  Tasks 
**l)  Duty  Assignment 
("A  Job  Activity 
#W  Type  of  Work  Environment 


